Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup

Knowledge distillation (KD) has emerged as an essential technique not only for model compression, but also other learning tasks such continual learning. Given the richer application spectrum and potential online usage of KD, knowledge efficiency becomes a pivotal component. In this work, we study little-explored important topic. Unlike previous works that focus solely on accuracy student network, attempt to achieve harder goal – obtain performance comparable conventional KD with lower computation cost during transfer. To end, present UNcertainty-aware mIXup (UNIX), effective approach can reduce transfer by 20% 30% yet maintain or even better than KD. This is made possible via uncertainty sampling novel adaptive mixup select informative samples dynamically over ample data compact in these samples. We show our inherently performs hard sample mining. demonstrate applicability improve various existing approaches reducing their queries teacher network. Extensive experiments are performed CIFAR100 ImageNet. Code available at https://github.com/xuguodong03/UNIXKD.